Computer-Implemented Methods and Systems for Testing Online Systems and Content

ABSTRACT

Computer-implemented methods and systems are provided for scanning web sites and/or parsing web content, including for testing online opt-out systems and/or cookies used by online systems. In accordance with one implementation, a computer-implemented method is provided for testing an opt-out system associated with at least one advertising system that uses cookies. The method includes transmitting a first request to an opt-out system, wherein the first request corresponds to a first test for testing at least one of the opt-out system and an advertising system; receiving a first stream sent in response to the first request; determining a first outcome of the first test based on the first stream; and generating a report based on the first outcome.

RELATED APPLICATION(S)

This application claims the benefit of priority of U.S. ProvisionalPatent Application No, 61/285,563, filed Dec. 11, 2009, which isincorporated by reference herein in its entirety.

BACKGROUND

1. Technical Field

The present disclosure generally relates to the field of automatedmonitoring technologies and techniques. More particularly, and withoutlimitation, the disclosure relates to computer-implemented methods andsystems for scanning web sites and/or parsing web content, including fortesting online opt-out systems and/or online content.

2. Background

As greater numbers of people use the World Wide Web for communication,commerce, and other daily activities, they generate larger and largervolumes of traffic over the Internet. Because the benefits ofcommercializing the Internet can be tremendous, businesses increasinglytake advantage of this traffic by advertising their products or servicesonline. These advertisements may appear in the form of leasedadvertising space (e.g., “banners”) on content websites, which areoperated by “publishers” or “advertising networks” who control thewebsite content and the availability and cost of the advertising spaceor “ad inventory.”

In most oases, there is a need to keep track of the number of adimpressions, and/or the quantity of click and conversion events relatedto advertisements. An tracking system is, therefore, necessary to keeptrack of ad-related events. An advertiser may also have a marketing planthat identifies certain types of people as being target audience membersfor a given product or service. For example, the advertiser may wish tospend money only on users having certain demographics or personalinterests. Alternatively, advertisers may be unsure of which people aremost likely to respond to a given product, service, or advertisement.Therefore, advertisers may wish to obtain very specific informationabout the types of consumers viewing various types of web sites andresponding to their advertisements. In some cases, advertisers may bewilling to spend more money per impression, dick, or conversion based onknown information about those users interacting with the advertisements.As a result, publishers of content websites and/or facilitators of thirdparty advertising networks (i.e., owners or operators of “advertisingsystems”) may wish to obtain as much information as possible aboutconsumers and other users browsing between web pages associated with anadvertising network.

In order to implement tracking systems and/or obtain information forproviding targeted ads, advertising systems may utilize tracking orbrowser cookies. In general, cookies comprise small sets of data thatcan be stored on users' computers by web browsers. Cookies may beimplemented using so-called HTTP or web cookies, as well as other knowntypes of cookies such as Flash cookies, “Evercookies,” “BrowserFingerprinting,” etc. When a user's computer visits a website or sends arequest for a file such as a banner ad, an advertising system maytransmit instructions to the user's web browser to store a cookie. Thecookie is stored for a specified time and returned whenever the user'scomputer makes a subsequent visit to the website or another website onthe same advertising network or system. The cookie may include one ormore name-value pairs that specifies information, such as a user'spreferences or browsing history. This information may allow theadvertising system to track ad-related events and/or display targetedads to the user based on collected information.

Cookies are also be used by online merchants and web service providersto support complex interactions and provide a better browsing experiencefor Internet users. For example, cookies may be used to facilitate auser log-in or provide information on the contents of an electronicshopping cart. Cookies can also be used to store registrationinformation or secure information. As with cookies used by advertisingsystems, cookies used by online merchants and web service providers canbe stored for a specified time and returned during subsequent requestsby a user's web browser.

For privacy or other reasons, an individual who does not want to receivetargeted or behavioral advertising may elect to “opt-out,” In general,opting-cut is a process by which a user may avoid receiving, forexample, targeted advertisements, including those initiated byinformation collected from cookies or other data sources. Users mayenroll in an opt-out system through various means, including a privacypolicy page or an opt-out page of a website. Some websites, such ashttp://www.networkadvertising.org, enable users to selectively opt-outof targeted ad programs for one or more advertising networks or systems.

In some cases, opt-out system failures may occur that improperly preventusers from being opted-out. The owners or operators of the system thatprovide cookies may not be aware of opt-out system failures until theyare alerted by a user or a third-party group. Efforts to manually ensurethat an opt-out system operates properly may be unrealistic due to thelarge number of contingencies and possible failures that need to bechecked on a continuous basis. Such efforts may require an enormousamount of resources. Furthermore, such efforts may not allow the ownersor operators of an advertising system to respond effectively if afailure were to be discovered.

In view of the foregoing, there is a need for automated methods andsystems that can accurately test online opt-out systems. There is also aneed for such methods and systems, where the testing of an online op-outsystem can be conducted on a continuous basis for one or more systemsand for a wide variety of contingencies and possible failures. Moreover,there is a need for improved methods and systems for testing onlineop-out systems that provide reporting features that enable owners oroperators of advertising systems to respond effectively when a failureis detected.

The herein disclosed embodiments are directed to achieving one or moreof the above-referenced goals, by providing methods and systems foranalyzing and testing online opt-out systems.

SUMMARY

In one exemplary embodiment, the present disclosure is directed to acomputer-implemented method for testing an online opt-out systemassociated with at least one advertising system that uses cookies. Themethod may include, for example, transmitting, by a test system, a firstrequest to an opt-out system, wherein the first request corresponds to afirst test for testing the opt-out system; receiving, by the testsystem, a first stream sent from the opt-out system in response to thefirst request; determining a first outcome of the first test byanalyzing contents of the first stream; transmitting, by the testsystem, a second request to the opt-out system, wherein the secondrequest corresponds to a second test for testing an advertising systemthat uses cookies; receiving, by the test system, an opt-out cookie sentfrom the advertising system; determining a second outcome of the secondtest by analyzing values of the opt-out cookie; and generating a reportbased on at least one of the first outcome and the second outcome.

In another exemplary embodiment, the present disclosure is directed to acomputer-implemented method for testing an opt-out system associatedwith at least one advertising system that uses cookies. The method mayinclude, for example, transmitting, by a test system, a first request toan opt-out system, wherein the first request corresponds to a first testfor testing at least one of the opt-out system and an advertising systemthat uses cookies; receiving, by the test system, a first stream sent inresponse to the first request; determining a first outcome of the firsttest based on the first stream; and generating a report based on thefirst outcome.

In another exemplary embodiment, the present disclosure is directed to acomputer-implemented method for receiving, by an opt-out system, a firstrequest from a test system, wherein the first request corresponds to afirst test for testing the opt-out system that is associated with atleast one advertising system that uses cookies; processing, by theopt-out system, the first request; transmitting, by the opt-out system,a first stream to the test system, wherein the first stream comprises anoutput of the processing; and transmitting, by the opt-out system,instructions to an advertising system to create an opt-out cookie and totransmit the opt-out cookie to the test system, wherein the test systemdetermines an outcome of the first test based on at least one of thefirst stream and the opt-out cookie.

In another exemplary embodiment, the present disclosure is directed to atest system for testing an online system associated with at least oneadvertising system that uses cookies. The test system may include, forexample, a storage configured to store at least one object for a firsttest for testing at least one of the online system and the advertisingsystem; and a processor configured to: transmit a first request to atleast one of the online system and the advertising system, wherein thefirst request corresponds to the first test, receive a first stream sentin response to the first request, determine a first outcome of the firsttest based on the first stream, and generate a report based on the firstoutcome.

In another exemplary embodiment, the present disclosure is directed to acomputer-implemented method for testing an online system that usescookies. The method may include, for example, generating a first requestto the online system, wherein the first request corresponds to a testfor testing at least one of an op-out condition and a cookie used by theonline system; receiving, by a server system, a first stream sent inresponse to the first request; determining a first outcome of the firsttest based on the first stream; and generating a report based on thefirst outcome.

Additional objects and advantages will be set forth in part in thedescription which follows, and in part will be obvious from thedescription, or may be learned by practice of the embodiments of theinvention. For example, the objects and advantages may be realized andattained by means of the elements and combinations particularly pointedout in the appended claims.

It is to be understood that both the foregoing general description andthe following detailed description are exemplary and explanatory onlyand are not restrictive of embodiments thereof, as claimed.

BRIEF DESCRIPTION OF THE DRAWINGS

The accompanying drawings, which are incorporated in and constitute apart of this disclosure, illustrate several embodiments and aspects ofthe present disclosure, and together with the description, serve toexplain certain principles of the invention. In the drawings:

FIG. 1 is a block diagram illustrating an exemplary environment forimplementing methods and systems for testing an online op-out system,consistent with certain disclosed embodiments;

FIG. 2 is a block diagram of an exemplary test system, consistent with adisclosed embodiment;

FIG. 3 is a flowchart of an exemplary method for testing an opt-outsystem, consistent with a disclosed embodiment;

FIG. 4 is a flowchart of another exemplary method for testing an opt-outsystem, consistent with a disclosed embodiment;

FIG. 5 is a flowchart of an exemplary method for performing an automatedhealing procedure in an opt-out system, consistent with a disclosedembodiment;

FIG. 6 is a table for illustrating testing the settings of an opt-outcookie on a test system, consistent with a disclosed embodiment; and

FIG. 7 is a table depicting an example of troubleshooting informationgenerated based on tests performed by a test system, consistent with adisclosed embodiment.

DESCRIPTION OF THE EMBODIMENTS

The following detailed description refers to the accompanying drawings.Wherever possible, the same reference numbers are used in the drawingsand the following description to refer to the same or similar parts.While several exemplary embodiments and features of the invention aredescribed herein, modifications, adaptations and other implementationsare possible, without departing from the spirit and scope of thedisclosed embodiments. For example, substitutions, additions, ormodifications may be made to the components illustrated in the drawings,and the exemplary methods described herein may be modified bysubstituting, reordering, or adding steps to the disclosed methods.Accordingly, the following detailed description does not limit theembodiments. Instead, the proper scope is defined by the appendedclaims.

FIG. 1 illustrates an exemplary environment 100 for implementing methodsand systems for testing an opt-out system, consistent with certainembodiments of the present disclosure. Environment 100 may include auser workstation 110, an opt-out system 130, an advertising system 140,a test system 150, and a configuration system 160, all disposed incommunication with a network 120. Each one of user workstation 110,opt-out system 130, advertising system 140, test system 150, andconfiguration system 160 may include at least one processor (e.g., acentral processing unit (“CPU”)), a memory, a modem, an input device(e.g., keyboard, mouse, etc.), an output device (e.g., monitor), and atleast one of internal storage (e.g., hard drive) and/or external storage(e.g., database on external servers, disk, flash memory, etc.). Network120 may be any combination of networks, for example, an intranet orprivate network, and/or the Internet. Any combination of opt-out system130, advertising system 140, test system 150, and configuration system160 may be operated by one or more operators (e.g., advertisingnetwork).

User workstation 110 may include, for example, a personal computer, alaptop, a handheld computer, a personal digital assistant (“PDA”), amobile device, a single server (or a component thereof), or any othercomputing platform capable of running a web browser 111 and receivingand storing cookies. Although only a single user workstation 110 isillustrated in FIG. 1, the exemplary environment 100 may include aplurality of such user workstations 110, each of which is associatedwith a different or unique user.

A user at user workstation 110 may choose to opt-out from having his/herusage patterns and web preferences collected for purposes of targeted orbehavioral advertising programs implemented by, for example, advertisingsystem 140. As further described below, a user at user workstation 110may opt-out to maintain his/her privacy by enrolling in an opt-outsystem, such as opt-out system 130. The opt-out system 130 may preventthe user from being part of one or more advertising systems' targetedadvertising programs, and also prevent user workstation 110 fromreceiving associated targeting or tracking cookies from advertisingsystem 140.

Advertising system 140 may include a personal computer, a single server(or a component thereof), multiple servers arranged as part of anintegrated or distributed system, a server farm, a network of servers,etc. Advertising system 140 may correspond to or be part of one or moreadvertising networks that provide online advertising. Advertising system140 may include one or more servers or components for transmitting andreceiving cookies. Additionally, advertising system 140 may include oneor more servers or components for delivering online advertisements, sucha plurality servers that are part of a content delivery network. Whileonly a single advertising system 140 is illustrated in FIG. 1, it willbe appreciated that the exemplary environment 100 may including aplurality of advertising systems 140 that are owned or operated by oneor more entities. Further, there may be one or more opt-out systems 130,wherein each opt-out system 130 is associated with one or moreadvertising systems 140 that use cookies.

Advertising system 140 may be configured to transmit targeting ortracking cookies to user workstations of users that have not opted-outfrom receiving such cookies. These and other types of cookies may betransmitted when a user visits and browses one or more websites in, forexample, an advertising network. As described above, cookies stored onnon-opted-out users' workstations 110 may allow the advertising system140 to track online usage and user preferences to transmit targetedadvertising. In contrast, for users that have elected to opt-out fromsuch cookies, no such targeting or tracking cookies may be utilized and,as a result, a user's workstation 110 of an opt-out user may onlyreceive general or non-targeted advertisements.

To opt-out from receiving targeting or tracking cookies from one or moreadvertising systems 140, a user may open or initiate a web browser 111on user workstation 110. Through browser 111, the user may navigate to awebsite associated with opt-out system 130. Once connected throughnetwork 120, the website of opt-out system 130 may be displayed to theuser in browser 111, including web pages with various opt-out options.By way of example, the user may select elements on pages of the websitedisplayed in browser 111 to opt-out from receiving targetedadvertisements from one or more advertising systems 140. For instance, alist of different advertising systems or networks may be displayed tothe user in browser 111. Additionally, or optionally, each listedadvertising system or network may have a corresponding status. Thestatus may indicate what kind of cookie (e.g., active cookie, opt-outcookie, no cookie), if any, user workstation 110 has previously receivedand stored. Based on the list, the user may elect to opt-out userworkstation 110 for one or more advertising systems 140 that usetargeting or tracking cookies.

Based on the opt-out options selected by a user, opt-out system 130 maythen send instructions to corresponding advertising system(s) 140 tocause each such system to create and send an opt-out cookie to userworkstation 110. In response, each advertising system 140 may create andsend an opt-out cookie to user workstation 110. Opt-out system 130 maythen display a message to the user in browser 111 whether userworkstation 110 was successfully opted-out from each advertising systemor network that the user selected.

After user workstation 110 receives an opt-out cookie from anadvertising system 140, the advertising system 140 may not be able totrack the activities of user workstation 110 when a user visits awebsite in the advertising network(s) associated with advertising system140. Whenever browser 111 visits a website in the advertising network ofadvertising system 140, user workstation 110 may send a request (such asfor a banner ad or other file) to advertising system 140. Along with therequest, browser 111 will transmit any previously stored cookiesassociated with advertising system 140. If user workstation 110 has anopt-out cookie corresponding to the advertising network of advertisingsystem 140, the opt-out cookie (e.g., with a value of “optout”) will betransmitted back to the advertising system 140. Advertising system 140may determine based on the opt-out cookie that user workstation 110 hasopted-out of targeted advertising programs and, therefore, determine notto use targeting or tracking cookies to track any activity or behaviorof the user of user workstation 110. As a result, only untargeted adsmay be transmitted to user workstation 110.

Test system 150 may include a personal computer, a single server (or acomponent thereof), multiple servers arranged as part of an integratedor distributed system, a server farm, a network of multiple servers,etc. Test system 150 may also be adapted to run or emulate differentworkstations and operating systems, as well as web browsers (InternetExplorer, Safari, Firefox, Chrome, etc.). When implemented, test system150 may automatically test whether an opt-out system is workingproperly. Among other things, test system 150 may send requests toopt-out system 130 and advertising system 140 through network 120. Testsystem 150 may also receive streams from opt-out system 130 andadvertising system 140 through network 120.

Consistent with embodiments of the present disclosure, test system 150may run a series of different tests periodically, at a predeterminedinterval (e.g., every hour). Settings of test system 150 may beconfigured before each test or series of different tests to simulate andanalyze, for example, certain environmental conditions. Environmentalconditions may include, by way of non-limiting examples, differentprivacy settings, different browsers, variations in computer host files(e.g., that may manage DNS settings), the presence of differentanti-spyware software, changing locations of test system 150, differentexisting cookies present on test system 150, etc.

Consistent with embodiments of the present disclosure, test system 150may simulate new environmental conditions after a series of differenttests are completed. In addition, test system 150 may repeat (rerun) theseries of different tests with the new environmental conditions. Testsystem 150 may also simulate a plurality of different environmentalconditions during each interval, and run a series of different tests foreach one of the environmental conditions.

In running a series of different tests, test system 150 may test thatone or more opt-out systems 130 are functioning properly. As describedabove, each opt-out system 130 may allow a user to opt-out fromreceiving targeted advertising and associated cookies from one or moreadvertising systems 140. Testing opt-out system 130 may ensure, amongother things, that a user can access and successfully complete or enrollin the opt-out process.

To conduct a test of opt-out system 130, test system 150 may send one ormore requests to opt-out system 130. Each request may represent one ormore requests in the form that a browser would send to opt-out system130. In other words, test system 150 may emulate requests from thebrowser of a user who is trying to opt-out through a website associatedwith opt-out system 130. Parameters of a request may differ based on thetest and the environmental conditions. Among other things, a test maycheck that the website of opt-out system 130 is functioning, check thatweb pages (e.g., opt-out pages, privacy policy page, opt-cut linkspages, etc.) of the website are operating properly, check that linksbetween the web pages of the website are operating properly, check thatnecessary elements to selectively opt-out from various advertisingnetworks or systems are present on the related web pages, check that ifa user were to select any elements (e.g., an element to opt-out from anadvertising network) that the opt-out system would respondappropriately, etc.

Test system 150 may receive a stream back from opt-out system 130 inresponse to the request(s) transmitted from test system 150. A streammay include one or more files, a text string, a character, a numericalvalue, a code, etc. Streams may be generated by opt-out system 130 afterprocessing requests received from test system 150. Test system 150 mayreceive multiple individual streams in response to a single request. Thenumber of streams received by test system 150 from each opt-out system130 may vary based on the number of requests that are transmitted. Inone embodiment, test system 150 may selectively receive or process onlythe streams test system 150 needs to determine an outcome of a test.Selectively receiving a stream may include determining whether thestream is necessary to determine the outcome of the test, and receivingor processing the stream only when the determination is positive.

Test system 150 may analyze a stream to identify relevant content withinthe stream that is desired to determine the outcome of a test. Testsystem 150 may perform any desired analysis and/or measurement of theidentified content to determine the outcome of a test. For example, testsystem 150 may determine that a website/web-page associated with opt-outsystem 130 is functioning when a stream is received in the form of aspecific response, such as an HTTP 200 OK result code, which indicatesindividually or with information provided in the response that the HTTPrequest has succeeded.

In accordance with embodiments of the present disclosure, test system150 may perform one or more tests, as part of a series of differenttests to determine whether opt-out system 130 or advertising system 140properly recognizes the cookies stored on test system 150. Test system150 may create different types of cookies (e.g., advertising networkcookie, opt-out cookie, no cookie). Test system 150 may send a requestto opt-out system 130 for opt-out system 130 to identify the types ofcookies stored on test system 150. Test system 150 may receive a streamfrom opt-out system 130 including identification of a type of cookiestored on test system 150 for a particular advertising network oradvertising system 140. Test system 150 may also test whether opt-outsystem 130 correctly identifies the type of cookies stored on testsystem 150 on a website associated with opt-out system 130. Test system150 may further test that the correct indicator for identifying a typeof cookie is properly displayed or otherwise indicated on a websiteassociated with opt-out system 130. In accordance with an embodiment,test system 150 may test that the indicator is properly displayed byrunning a checksum on an image file of the indicator.

In accordance with additional embodiments, test system 150 may performone or more tests to determine whether opt-out system 130 properlyinteracts with each corresponding advertising system 140. Test system150 may also perform one or more tests to determine whether eachadvertising system 140 is responding correctly to the interaction withopt-out system 130. To perform a test relating to the interaction ofopt-out system 130 and an advertising system 140, test system 150 maysend a request for opt-out system 130 to prompt an advertising system140 to run a script. Thereafter, the advertising system 140 may run thescript to create an opt-out cookie and send the opt-out cookie to testsystem 150. The request simulates a selection (by a user) to opt-outtest system 150 from the advertising network associated with theadvertising system 140.

Test system 150 may send a request to opt-out system 130 to test thatonly the advertising systems 140 corresponding to advertising networksthat are selected for the opt-out process and that are prompted tocreate opt-out cookies (e.g., if a user selects to opt-out only out ofreceiving cookies from a particular advertising network, only theadvertising system corresponding to that advertising network is promptedto run a script to create an opt-out cookie). Test system 150 may send arequest to an advertising system 140 to test that the correct script iscalled to create a cookie. The advertising system 140 may then run ascript to create an opt-out cookie for test system 150 in response to aprompt or instruction from opt-out system 130.

In accordance with yet additional embodiments, test system 150 may thenrun tests to determine that test system 150 receives a proper opt-outcookie from each advertising system 140 based on the requests that testsystem 150 sent to opt-out system 130. After each advertising system 140creates the opt-out cookie and sends the opt-out cookie to the testsystem 150, test system 150 may analyze the cookie to determine thatvalues of the cookie are correct. For example, test system 150 may run atest to determine that the lifespan of the opt-out cookie is consistentwith predetermined guidelines (e.g., no shorter than 3 or 5 years). Testsystem 150 may run other tests to determine that other values associatedwith an opt-out cookie are correct, such as: there is no unique IDparameter to track the cookie or personal information related to theuser, all ID values are set to a dummy value (e.g., representing an“optout” value), tracking value is set to a predetermined tracking valuefor opt-out cookies (e.g., 0), etc. Test system 150 may also performtests based on the type or form of opt-out cookie (e.g., HTTP cookie, aFlash cookie, “Evercookie,” “Browser Fingerprinting,” etc.). Test system150 may also run tests to determine whether other cookies, besidesopt-out cookies, transmitted from advertising system 140 have propervalues. For example, test system 150 may run a test to determine thatthe lifespan of an ad cookie is no longer than a predetermined amount oftime.

Test system 150 may run tests to determine whether the opt-out cookiereceived from an advertising system 140 is set correctly on test system150 depending on what, if any, cookie corresponding to the advertisingsystem 140 is stored on test system 150 (see, e.g., the exemplaryembodiment of FIG. 6). Test system 150 may repeat tests to determinewhether opt-out system 130 correctly identifies the type of cookie(e.g., opt-out) on test system 150 and that opt-out system 130 displaysthe correct indicator on the website of opt-out system 130.

According to still further embodiments, test system 150 may test whetheran opt-out cookie set on test system 150 functions properly. Test system150 may send a request to a website that is connected (directly orindirectly) to an advertising network corresponding to an advertisingsystem 140. To perform the test, test system 150 may send a requestdirectly to the advertising system 140. The advertising system 140 mayrespond to the request from test system 150. Test system 150 may run atest to determine whether an opt-out cookie is transmitted to theadvertising system 140 with or in response to the request.

After test system 150 determines an outcome of one or more individualtests, test system 150 may generate a report based on the outcome ofeach test. Test system 150 may log the report in a database or inanother form on, for example, an internal storage of test system 150 ora storage device external to test system 150. Additionally, oralternatively, test system 150 may transmit a report through email orother suitable communication means. Reports may be sent to owners oroperators of an opt-out system or advertising network or system.Additionally, or alternatively, reports may be transmitted to a thirdparty or members of a specific action group associated with anadvertising network or advertising system 140. The members of a specificaction group may include one or more persons responsible for fixing anyfailures identified in a report for a particular test. Test system 150may also transmit such reports when problems or failures are identifiedby the outcome of a test (i.e., the outcome of the test is negative).

In accordance with additional embodiments, test system 150 may waituntil all tests within a series of different tests are completed in aninterval before generating one or more reports. Test system 150 may thenorganize the information from the reports into different emails forspecific entities or action groups. Test system 150 may also organizeemails for individual members of the specific action groups that compileinformation from reports based on an individual member's membership indifferent action groups. Test system 150 may determine a level ofseriousness of any failures or problems identified within a series ofdifferent tests. The level of seriousness may be based on the number ofproblems identified and for what specific tests the problems areidentified. When a levels of seriousness is above a predeterminedthreshold, test system 150 may send an alert email to, for example, aprivacy group or a specific individual (e.g., a high level manager) whois not a member of any specific action group.

Test system 150 may use configuration system 160 to perform an automatedhealing procedure to fix any failures or problems that are detectedduring any tests. As noted above, problems may be detected when theoutcome of a test is negative. Test system 150 may converttroubleshooting information in one or more of the generated reports intomachine-readable instructions. Test system 150 may transmit themachine-readable instructions to configuration system 160.Alternatively, test system 150 may first compile the machine-readableinstructions and then send the compiled machine-readable instructions toconfiguration system 160.

Configuration system 160 may communicate with each one of test system150, advertising system 140, and/or opt-out system 130, either directlyor through network 120. Configuration system 160 may include a personalcomputer, a single server (or component thereof), multiple serversarranged in a network or as a server farm, etc. Configuration system 160may be implemented independently or as part of test system 150.Configuration system 160 may manage each advertising system 140 andopt-out system 130. Configuration system 160, based on themachine-readable instructions or the compiled machine-readableinstructions, may perform one or more of the following steps on at leastone of opt-out system 130 and advertising system 140: (1) reboot aserver; (2) reconfigure a server; (3) reformat a server; and (4) removea server from rotation. Test system 150 may rerun a series of differenttests after any one of the foregoing steps is performed by configurationsystem 160 to make sure that the problem or failure has been corrected.

FIG. 2 depicts an exemplary test system 150, consistent with embodimentsof the present disclosure. As illustrated in FIG. 2, test system 150 mayinclude one or more testing objects 210, a testing engine 220, a reportgenerator 230, and an automated troubleshooter 240. As will beappreciated, the above components of test system 150 may be implementedthrough any suitable combination of hardware, software, and/or firmware.

Testing objects 210 may include one or more independent objects. Eachone of testing objects 210 may correspond to a different test performedby test system 150. Each one of testing objects 210 may specify requestsdesired to be sent to either opt-out system 130 or advertising system140 from test system 150, to perform each particular test and/or theparameters desired to be set for the requests. Each one of testingobjects 210 may also specify the streams desired to be received fromeither opt-out system 130 or advertising system 140 (e.g., how thestreams can be identified) and the important content of the streams.Each one of testing objects 210 may also specify how the content may beanalyzed and measured to determine an outcome for each particular test.Each one of testing objects 210 may be run through testing engine 220.

Testing engine 220 may change the settings of test system 150 beforeexecuting the tests corresponding to each one of testing objects 210.Testing engine 220 may change settings of test system 150 for eachindividual test, as needed. Testing engine 220 may also change settingsof test system 150 before executing each one of the tests correspondingto each one of testing objects 210, execute each one of the tests,change settings of test system 150 to new settings, re-execute each oneof the tests again, etc. Testing engine 220 may compile code needed toexecute code corresponding to testing objects 210. Testing engine 220may transmit a request corresponding to each one of testing objects 210to either opt-out system 130 or advertising system 140. Testing engine220 may also receive a stream corresponding to the request from opt-outsystem 130 or advertising system 140. Testing engine 220 may receive thestream and identify and analyze the contents of the stream. Testingengine 220 may also determine an outcome of each test.

Report generator 230 may compile a report based on a test outcome and/ortroubleshooting information associated with each one of testing objects210. The troubleshooting information may include an outcome for a test,information about a configuration of the tests, network traceinformation, function(s) that declared an error, various statistics,etc. Report generator 230 may determine who needs to be alerted about aparticular compiled report. Report generator 230 may generate an emailwith the report. Report generator 230 may transmit the email with thecompiled report to a single specific action group, multiple specificaction groups, a privacy group, specific point of contact, etc.

Automated troubleshooter 240 may determine whether any failures orproblems are discovered as a result of the tests (i.e., whether any ofthe outcomes are negative). If problems are detected, automatedtroubleshooter 240 may convert the troubleshooting information from oneor more reports into machine-readable instructions. Automatedtroubleshooter 240 may compile the machine-readable instructions.Automated troubleshooter 240 may call configuration system 150 to actupon the troubleshooting information. Automated troubleshooter 240 maytransmit the machine-readable instructions to configuration system 160.

FIG. 3 depicts a flowchart of exemplary method 300 for testing anopt-out system, consistent with embodiments of the present disclosure.Method 300 may be executed using the components of environment 100illustrated in FIG. 1. According to method 300, at step 310, test system150 may test opt-out system 130. Test system 150 may test opt-out system130 by conducting one or more different tests (e.g., a test to determinewhether an opt-out website is functioning, a test to determine whetherelements to opt-out are present on an opt-out website, a test todetermine whether the elements respond properly, a test to determinewhether opt-out system 130 interacts properly with test system 150, atest to determine whether opt-out system 130 interacts properly witheach advertising system 140, etc.). As noted above, each one of thedifferent tests may correspond to one of testing objects 210 depicted inFIG. 2.

At step 320, test system 150 may test advertising system 140. The testsfor advertising system 140 may include determining whether advertisingsystem 140 properly runs a script to create an opt-out cookie and sendsthe opt-out cookie to test system 150. At step 330, test system 150 mayanalyze the opt-out cookie. Test system 150 may analyze the opt-outcookie to determine that the opt-out cookie includes proper values, asdescribed above.

At step 340, test system 150 may determine whether the opt-out cookie isproperly set on test system 150. At step 350, test system 150 may test aresponse of advertising system 140 to make sure that the opt-outinstructions are being adhered to by the advertising system 140. Forexample, test system may send a request (such as a request for anadvertisement or other file) to test whether advertising system 140properly responds to an opt-out cookie set on test system 150. In oneembodiment, test system 150 transmits the opt-out cookie to advertisingsystem 140 as part of an HTTP request. Test system 150 then monitors anHTTP response from advertising system 140 to make sure, for example,that targeting or tracking cookies are not transmitted by advertisingsystem 140.

At step 360, test system 150 may generate a report based on the outcomeof one or more tests. As part of this step, test system 150 may log thereport on a database of test system 150 or on some other internal orexternal storage device. Additionally, test system 150 may transmit thereport through email or other communication means to appropriaterecipients. As noted above, the appropriate recipients may include atleast a specific action group. Step 360 may occur after each one ofsteps 310, 320, 330, 340, and 350, or after each one of the specifictests conducted within steps 310, 320, 330, 340, and 350.

Optionally, as part of the exemplary method of FIG. 3, a healingprocedure may be performed to correct detected problems or failures.More specifically, at step 370, test system 150 may execute an automatedhealing procedure for opt-out system 130. Step 370 may include, forexample, generating machine-readable instructions based on one or morereports, compiling the machine-readable instructions, and promptingconfiguration system 160 to correct any problems or failures identifiedin the one or more reports. Test system 150 may repeat one or more ofsteps 310, 320, 330, 340, 350, and 360, after each attempt byconfiguration system 160 to correct detected problems in step 370.

FIG. 4 depicts a flowchart of another exemplary method 400 for testingan opt-out system, consistent with embodiments of the presentdisclosure. Method 400 may be executed by test system 150 of FIG. 1.According to method 400, at step 410, test system 150 may be configuredbefore any tests are executed. Configuring test system 150 may includesimulating or emulating particular environmental conditions on testsystem 150, as detailed above.

At step 420, test system 150 may send a request, corresponding to one ofthe tests for opt-out system 130. At step 430, test system 150 mayreceive a stream from opt-out system 130 in response to the request.Opt-out system 130 may send multiple streams in response to the request.At step 430, test system 150 may selectively determine which one or morestreams to receive from opt-out system 130 based on the particular testthat is being performed.

At step 440, test system 150 may analyze the received stream. Analyzingthe stream, at step 440, may include selecting content from the streamdesired to determine an outcome of a test, analyzing the selectedcontent by, for example, measuring values of the contents, and determinean outcome of a test based on the measurements. After step 440 iscomplete, test system 150 may repeat steps 420, 430, and 440 forsubsequent tests of opt-out system 130. Steps 420, 430, and 440 may beincluded in step 310 of method 300 depicted in FIG. 3.

At step 450, test system 150 may receive an opt-out cookie fromadvertising system 140. Before step 450, test system 150 may send arequest to opt-out system 130 to opt-out of targeted advertisingprograms of particular advertising networks or advertising systems. Therequest may prompt each corresponding advertising system 140 to run ascript to create the opt-out cookie and send the opt-out cookie to testsystem 150. As a result, at step 450, test system 150 may receive theopt-out cookie. At step 460, test system 150 may analyze the receivedopt-out cookie to determine whether values in the cookie are proper.When the received cookie is supposed to be an opt-out cookie (based onthe request originally sent by test system 150 to opt-out system 130),test system 150 may analyze whether the values in the cookie are properfor an opt-out cookie.

At step 470, test system 150 may determine whether the received cookieis properly set on test system 150. After step 440 is complete, testsystem 150 may repeat steps 450, 460, and 470 for subsequent tests. Forexample, for an alternative test, test system 150 may prompt advertisingsystem 140 to either send an opt-out cookie to test system 150 orrequest a normal cookie from test system 150. After all tests arecomplete, test system 150 may restart at step 410. At step 410, testsystem 150 may re-configure the settings of test system 160 to set newenvironmental conditions. After the new environmental conditions areset, test system 150 may repeat steps 420, 430, 440, 450, 460, and 470.

FIG. 5 is a flowchart of an exemplary method for performing an automatedhealing procedure after test system 150 determines that automatedhealing is necessary. Test system 150 may determine that automatedhealing is desired after conducting one or more automated tests ofopt-out system 130 and discovering one or more failures/problems one ormore outcomes of the tests are negative). Method 500 may be executed bytest system 150 and/or configuration system 160. According to method500, at step 510, test system 160 may convert troubleshootinginformation, detailing a problem detected during one of the testsperformed by test system 150, into machine-readable instructions. Atstep 520, test system 150 may compile the machine-readable instructions.At step 530, test system 150 may transmit the compiled machine-readableinstructions to configuration system 160.

At step 540, configuration system 160 may reboot at least one server (orset of servers) of opt-out system 130 and advertising system 140. In thedescription of steps 540, 550, 560, and 570, opt-out system 130 andadvertising system 140, may refer to only one individual physical server(or set of servers) that is responsible for the detected problem (theindividual physical server(s) may be described by troubleshootinginformation). Alternatively, multiple individual physical systems maycomprise opt-out system 130 and cookie system 140. Rebooting a servermay include first shutting the server down and then powering the serverback up. Instead of or in addition to step 540, configuration system 160may optionally alert test system 150 to re-execute one or more tests ofopt-out system 100. Configuration system 160 may alert test system 150to only re-execute a subset of tests to determine whether a previouslydiscovered problem has been corrected. If test system 150 determines,based on an outcome of a re-executed test, that a problem has not beencorrected by rebooting at step 540, configuration system may optionallyproceed with step 550 (after repeating steps 510, 520, and 530 andskipping step 540).

At step 650, configuration system 160 may optionally reconfigure atleast one server (or set of servers) of opt-out system 130 andadvertising system 140. Reconfiguring a server (or set of servers) mayinclude pushing down the most up-to-date configuration file on theserver(s). Reconfiguring may also include rebooting the server(s) afterthe most up-to-date configuration file is pushed down. After step 550,configuration system 160 may alert test system 150 to re-execute one ormore tests of opt-out system 100. If test system 150 determines, basedon an outcome of a re-executed test, that a problem has not beencorrected by reconfiguring at step 550, configuration system may proceedwith step 560 (after repeating steps 510, 520, and 530 and skippingsteps 540 and 550).

At step 560, configuration system 160 may optionally reformat at leastone server (or set of servers) of opt-out system 130 and advertisingsystem 140. After step 560, configuration system 160 may alert testsystem 150 to re-execute one or more tests of opt-out system 100. Iftest system 150 determines, based on an outcome of a re-executed test,that a problem has not been corrected by reformatting at step 560,configuration system may proceed with step 570 (after repeating steps510, 520, and 530 and skipping steps 540, 550, and 560).

At step 570, configuration system 160 may determine whether there is atleast one other individual physical server that correctly performs thesame role as the individual physical server responsible for a detectedproblem. For example, there may be at least one other server whenopt-out system 130 is comprised of two individual physical servers,where one individual physical server is responsible for the problem andthe other individual physical server correctly performs the same role.If there is at least one other server, configuration system 160 mayremove the individual physical server responsible for the problem fromrotation. Removing the individual physical server from rotation may notallow the server to act as part of opt-out system 130. If there is noother server that correctly performs the same purpose, automated healingmay not be successful when the problem is not corrected after steps 540,550, and 560. The problem may not be corrected until someone, forexample, from the specific action group, resolves the problem inresponse to an alert through email (with a report).

FIG. 6 is a table 600 that illustrates exemplary testing of settings ofan opt-out cookie on test system 150, consistent with embodiments of thepresent disclosure. Table 600 shows examples of what cookies may bepresent on test system 150 before the setting of an opt-out cookie aspart of a test. Test system 150 may be configured to have no cookies(e.g., Present Cookie=None) for an advertising network associated withadvertising system 140 before a test is conducted by test system 150.Test system 150 may receive an opt-out cookie corresponding toadvertising system 140. Test system 150 may determine that the opt-outcookie is properly set on test system 150. Thereafter, test system 150may send a request to a website associated with advertising system 140to test that the opt-out cookie is properly set.

Test system 150 may also be configured to have a cookie corresponding toadvertising system 140 before a test is conducted by test system 150.Test system 150 may receive an opt-out cookie corresponding toadvertising system 140. Test system 150 may determine that test system150 properly overrides the advertising system cookie with the opt-outcookie by deleting the advertising system cookie and setting the opt-outcookie.

Test system 150 may also be configured to already have an opt-out cookiefor advertising system 140 set on test system 150 before a test isconducted by test system 150. Test system 150 may receive anotheropt-out cookie for advertising system 140. Test system 150 may determinethat test system 150 already has an existing opt-out cookie, that theopt-out cookie is not overridden, that test system 150 does not need thenewly received opt-out cookie, and that the existing opt-out cookiecontinues to be properly set.

FIG. 7 is a table depicting an example of troubleshooting information710 for tests performed by test system 150. Troubleshooting information710 may be compiled into a report by test system 150. As illustrated inFIG. 7, troubleshooting information 710 may include, for example, aconfiguration of a test 711, network trace information 712, anidentification of a function of the test that determined a problem 713,and relevant statistics 714. Configuration of test 711 may include oneor more of the following: an identification of the test during which aproblem was identified, settings of test system 150 at a time when thetest was executed, cookies on the test system 150 at the time when thetest was executed, etc. Network trace information 712 may include adetailed history of the test: what information was transmitted from testsystem 150, to what servers) the information was transmitted from testsystem 150, what information was transmitted to test system 150, fromwhat server(s) the information was transmitted to test system 150, howany information was transmitted, etc. Examples of network traceinformation 712 may include information about requests, streams, andtransmission information related to the test. Network trace information712 may include physical identifications of the different servers andnetwork routes used during a test. Relevant statistics 714 may includeother information about the execution of the test, history of theexecution of the test in the past, etc.

The foregoing description has been presented for purposes ofillustration. It is not exhaustive and is not limiting to the preciseforms or embodiments disclosed. Modifications and adaptations will beapparent to those skilled in the art from consideration of thespecification and practice of the disclosed embodiments. For example,the described implementations include software, but systems and methodsconsistent with the disclosed embodiments may be implemented as acombination of hardware and software or in hardware alone. Examples ofhardware include computing or processing systems, including personalcomputers, s, laptops, mainframes, micro-processors and the like.Additionally, although aspects are described for being stored in memory,one skilled in the art will appreciate that these aspects can also bestored on other types of computer-readable media, such as secondarystorage devices, for example, hard disks, floppy disks, or CD-ROM, orother forms of RAM or ROM.

Computer programs based on the written description and methods arewithin the skill of an experienced developer. The various programs orprogram modules can be created using any of the techniques known to oneskilled in the art or can be designed in connection with existingsoftware. For example, program sections or program modules can bedesigned in or by means of C#, Java, C++, HTML, XML, or HTML withincluded Java applets. One or more of such software sections or modulescan be integrated into a computer system or existing e-mail or browsersoftware.

Moreover, while illustrative embodiments have been described herein, thescope of thereof includes any and all embodiments having equivalentelements, modifications, omissions, combinations (e.g., of aspectsacross various embodiments), adaptations and/or alterations as would beappreciated by those in the art based on the present disclosure. Forexample, while embodiments of the present disclosure have been describedwith respect to the testing of opt-out systems related to advertisingnetworks or advertising systems, embodiments of the present disclosuremay also be implemented for other types of opt-out systems or onlinesystems and content. The disclosed embodiments can also be adapted fordirect marketing systems and email advertising systems that employ, forexample, mailing or solicitation lists. It will be appreciated, forexample, the disclosed embodiments may be adapted for a wide variety ofopt-out systems, including those pertaining to networks or systems thatuse cookies or behavioral/targeting information concerning users (e.g.,merchant systems).

The limitations in the claims are to be interpreted broadly based on thelanguage employed in the claims and not limited to examples described inthe present specification or during the prosecution of the application,which examples are to be construed as non-exclusive. Further, the stepsof the disclosed methods may be modified in any manner, including byreordering steps and/or inserting or deleting steps. It is intended,therefore, that the specification and examples be considered asexemplary only, with a true scope and spirit being indicated by thefollowing claims and their full scope of equivalents.

1.-29. (canceled)
 30. A computer-implemented method for testing anopt-out system for opting-out of cookies, the method comprising thefollowing operations performed by a test system including at least oneprocessor: sending, to the opt-out system, a request to opt-out ofcookies, the request causing the opt-out system to instruct anadvertising system to create an opt-out cookie; receiving the opt-outcookie from the advertising system; and determining whether the opt-outcookie from the advertising system includes a predetermined value. 31.The computer-implemented method of claim 30, further comprising:analyzing an environmental condition before sending the request toopt-out of cookies.
 32. The computer-implemented method of claim 30,wherein sending the request to opt-out of cookies comprises: emulatingsending the request from an Internet browser of a user who desires toopt-out of cookies through a website associated with the opt-out system.33. The computer-implemented method of claim 32, wherein the request toopt-out of cookies comprises at least one of a parameter for verifyingthat the website of the opt-out system is functioning to allow the userto enroll in an opt-out process.
 34. The computer-implemented method ofclaim 30, further comprising: receiving, from the opt-out system, astream of data generated by the opt-out system in response to therequest to opt-out of cookies; and processing the stream of data. 35.The computer-implemented method of claim 34, further comprising:determining, before receiving the stream of data, whether the stream ofdata is necessary to determine an outcome of a test; and receiving thestream of data when the stream of data is necessary to determine theoutcome of the test.
 36. The computer-implemented method of claim 35,further comprising: analyzing the stream of data to identify contentfrom the stream of data that is necessary to determine the outcome ofthe test.
 37. The computer-implemented method of claim 30, furthercomprising: creating a plurality of types of cookies; and storing, inthe test system, the plurality of types of cookies, wherein the requestto opt-out cookies comprises a request for the opt-out system toidentify the types of the cookies stored in the test system, and whereinthe operations further comprise: receiving a stream of data from theopt-out system including information indicating identification of thetypes of the cookies stored in the test system; and determining whetherthe information received from the opt-out system correctly identifiesthe types of the cookies.
 38. The computer-implemented method of claim37, wherein determining whether the information correctly identifies thetypes of the cookies comprises: determining whether a predeterminedindicator is indicated on a website associated with the opt-out system.39. The computer-implemented method of claim 30, wherein the requestcauses the opt-out system to instruct the advertising system to run ascript to create the opt-out cookie and to send the opt-out cookie tothe test system.
 40. A test system for testing a computer systemconnected with the test system, comprising: a storage medium that storesa set of instructions; and at least one processor that executes the setof instructions to perform operations for testing the computer system,the operations comprising: sending a request to the computer system, therequest relating to opting-out of cookies; receiving a response from thecomputer system; analyzing the response; and determining an outcome ofthe testing based on a result of analyzing the response.
 41. The testsystem of claim 40, further comprising: a report generator configured togenerate a report based on the outcome.
 42. The test system of claim 41,wherein the report generator is further configured to determine arecipient for receiving an alert about the report.
 43. The test systemof claim 41, further comprising a troubleshooter configured to:determine whether the outcome is negative; and convert troubleshootinginformation from the report into machine-readable instructions when theoutcome is negative.
 44. The test system of claim 40, wherein theresponse includes an opt-out cookie, and wherein the at least oneprocessor further executes the set of instructions to analyze theopt-out cookie to determine whether the opt-out cookie includes apredetermined value.
 45. The test system of claim 44, wherein the atleast one processor further executes the set of instructions todetermine whether the opt-out cookie is properly set on the test system.46. The test system of claim 45, wherein the computer system includes anadvertising system, and wherein the opt-out cookie is generated by theadvertising system.
 47. The test system of claim 46, wherein the atleast one processor further executes the set of instructions to overridea cookie generated by the advertising system with the opt-out cookie.48. The test system of claim 40, wherein the computer system includes anadvertising system, and the response includes a first opt-out cookiegenerated by the advertising system; and the at least one processorfurther executes the set of instructions to determine whether the testsystem already includes a previously received opt-out cookie generatedby the advertising system, and to determine not to override thepreviously received opt-out cookie with the first opt-out cookie if thetest system already includes the previously received opt-out cookie. 49.A non-transitory computer-readable medium that stores a set ofinstructions, which when executed by at least one processor, cause theat least one processor to perform operations for testing an opt-outsystem for opting-out of cookies, the operations comprising: sending, tothe opt-out system, a request to opt-out of cookies, the request causingthe opt-out system to instruct an advertising system to create anopt-out cookie; receiving the opt-out cookie from the advertisingsystem; and determining whether the opt-out cookie from the advertisingsystem includes a predetermined value.